We present an autoregressive end-to-end learning approach for probabilistic shaping on nonlinear fiber channels. Our proposed scheme learns the joint symbol distribution and provides a 0.3-bits/2D achievable information rate gain over an optimized marginal distribution for dual-polarized 64-QAM transmission over a single-span 205 km link.

For the first time, multi-task learning is proposed to improve the flexibility of NN-based equalizers in coherent systems. A "single" NN-based equalizer improves Q-factor by up to 4 dB compared to CDC, without re-training, even with variations in launch power, symbol rate, or transmission distance.

In this paper, we investigate the use of the learned digital back-propagation (LDBP) for equalizing dual-polarization fiber-optic transmission in dispersion-managed (DM) links. LDBP is a deep neural network that optimizes the parameters of DBP using the stochastic gradient descent. We evaluate DBP and LDBP in a simulated WDM dual-polarization fiber transmission system operating at the bitrate of 256 Gbit/s per channel, with a dispersion map designed for a 2016 km link with 15% residual dispersion. Our results show that in single-channel transmission, LDBP achieves an effective signal-to-noise ratio improvement of 6.3 dB and 2.5 dB, respectively, over linear equalization and DBP. In WDM transmission, the corresponding $Q$-factor gains are 1.1 dB and 0.4 dB, respectively. Additionally, we conduct a complexity analysis, which reveals that a frequency-domain implementation of LDBP and DBP is more favorable in terms of complexity than the time-domain implementation. These findings demonstrate the effectiveness of LDBP in mitigating the nonlinear effects in DM fiber-optic transmission systems.

In this paper, we introduce a new nonlinear channel equalization method for the coherent long-haul transmission based on Transformers. We show that due to their capability to attend directly to the memory across a sequence of symbols, Transformers can be used effectively with a parallelized structure. We present an implementation of encoder part of Transformer for nonlinear equalization and analyze its performance over a wide range of different hyper-parameters. It is shown that by processing blocks of symbols at each iteration and carefully selecting subsets of the encoder's output to be processed together, an efficient nonlinear compensation can be achieved. We also propose the use of a physic-informed mask inspired by nonlinear perturbation theory for reducing the computational complexity of Transformer nonlinear equalization.

In this work, we propose to use various artificial neural network (ANN) structures for modeling and compensation of intra- and inter-subcarrier fiber nonlinear interference in digital subcarrier multiplexing (DSCM) optical transmission systems. We perform nonlinear channel equalization by employing different ANN cores including convolutional neural networks (CNN) and long short-term memory (LSTM) layers. We start to compensate the fiber nonlinearity distortion in DSCM systems by a fully connected network across all subcarriers. In subsequent steps, and borrowing from fiber nonlinearity analysis, we gradually upgrade the designs towards modular structures with better performance-complexity advantages. Our study shows that putting proper macro structures in design of ANN nonlinear equalizers in DSCM systems can be crucial for practical solutions in future generations of coherent optical transceivers.

Moreover, with the ever-increasing transmission bandwidth, nonlinearity becomes even more important [1]. Various digital signal processing (DSP) techniques have been proposed to minimize nonlinear effects [2]. Due to the universal approximation capability of neural networks (NNs), the NNs have recently been intensively studied for the optical channel post-equalization, because they can approximate the inverse optical channel transfer function with good accuracy and revert the nonlinear distortions. In particular, recurrent NNs (RNN) based equalizers have shown the best capability in equalizing nonlinear impairments as compared to the feed-forward NN types [3-5]. However, since the RNN structure has a feedback loop, it is not easily parallelizable.

In this paper, an unsupervised machine learning method for geometric constellation shaping is investigated. By embedding a differentiable fiber channel model within two neural networks, the learning algorithm is optimizing for a geometric constellation shape. The learned constellations yield improved performance to state-of-the-art geometrically shaped constellations, and include an implicit trade-off between amplification noise and nonlinear effects. Further, the method allows joint optimization of system parameters, such as the optimal launch power, simultaneously with the constellation shape. An experimental demonstration validates the findings. Improved performances are reported, up to 0.13 bit/4D in simulation and experimentally up to 0.12 bit/4D.

A new geometric shaping method is proposed, leveraging unsupervised machine learning to optimize the constellation design. The learned constellation mitigates nonlinear effects with gains up to 0.13 bit/4D when trained with a simplified fiber channel model.